Skip to main content

第 13 章：管理日誌與監控環境

Logging​

  • kubectl logs → API Server → kubelet → container logs
kubectl logs
kubectl logs $POD
kubectl logs $POD -c $CONTAINER
kubectl logs $POD --all-containers
kubectl logs --selector app=demo
kubectl logs -f $POD  # follow latest logs
kubectl logs $POD --tail 5   # get last 5 entries logs
  • 如果 api server 不可用，則需要手動去 container 所在節點查看 container Log
crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs $CONTAINER_ID

或者

tail /var/log/containers/$CONTAINER_NAME-$CONTAINER_ID

Accessing Log Data - Nodes​

Node 上有兩個關鍵的元件，一個是 kubelet ，ㄧ個是 kube-proxy

systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Wed 2024-01-10 07:54:41 UTC; 2h 11min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 5709 (kubelet)
      Tasks: 11 (limit: 4667)
     Memory: 36.7M
        CPU: 2min 651ms
     CGroup: /system.slice/kubelet.service
             └─5709 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --contain>

Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.074323    5709 kuberuntime_manager.go:1119] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup >
Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.074385    5709 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-5dd5756b68-rwbt8>
Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.087210    5709 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to se>
Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.087291    5709 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup n>
Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.087318    5709 kuberuntime_manager.go:1119] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup >
Jan 10 07:54:57 k8s-msr-1 kubelet[5709]: E0110 07:54:57.087379    5709 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-5dd5756b68-m5h7c>
Jan 10 07:55:09 k8s-msr-1 kubelet[5709]: I0110 07:55:09.533906    5709 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-flannel/kube-flannel-ds-n97mk" podStart>
Jan 10 07:55:09 k8s-msr-1 kubelet[5709]: I0110 07:55:09.555666    5709 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/coredns-5dd5756b68-rwbt8" podSta>
Jan 10 07:55:12 k8s-msr-1 kubelet[5709]: I0110 07:55:12.567640    5709 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/coredns-5dd5756b68-m5h7c" podSta>
Jan 10 09:51:53 k8s-msr-1 kubelet[5709]: I0110 09:51:53.055383    5709 scope.go:117] "RemoveContainer" containerID="12aa2cc38d2140c026544027735e10fe5e80c3aa02846a26485f5fe57fd694a6"
journalctl -u kubelet.service
Jan 10 07:54:24 k8s-msr-1 kubelet[5253]: I0110 07:54:24.278550    5253 server.go:203] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should a>
Jan 10 07:54:24 k8s-msr-1 kubelet[5253]: I0110 07:54:24.513047    5253 server.go:467] "Kubelet version" kubeletVersion="v1.28.0"
Jan 10 07:54:24 k8s-msr-1 kubelet[5253]: I0110 07:54:24.513092    5253 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jan 10 07:54:24 k8s-msr-1 kubelet[5253]: I0110 07:54:24.513567    5253 server.go:895] "Client rotation is on, will bootstrap in background"
Jan 10 07:54:24 k8s-msr-1 kubelet[5253]: E0110 07:54:24.519103    5253 certificate_manager.go:562] kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate>
journalctl -u kubelet.service | grep -i ERROR
Jan 10 07:53:51 k8s-msr-1 kubelet[5023]: E0110 07:53:51.957038    5023 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Jan 10 07:54:02 k8s-msr-1 kubelet[5151]: E0110 07:54:02.077945    5151 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Jan 10 07:54:12 k8s-msr-1 kubelet[5200]: E0110 07:54:12.301146    5200 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
journalctl -u kubelet.service --since today --no-pager
Jan 10 07:53:51 k8s-msr-1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 10 07:53:51 k8s-msr-1 systemd[1]: kubelet.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Jan 10 07:53:51 k8s-msr-1 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Jan 10 07:53:51 k8s-msr-1 systemd[1]: kubelet.service: Deactivated successfully.
Jan 10 07:53:51 k8s-msr-1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 10 07:53:51 k8s-msr-1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 10 07:53:51 k8s-msr-1 kubelet[5023]: E0110 07:53:51.957038    5023 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Jan 10 07:53:51 k8s-msr-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 07:53:51 k8s-msr-1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 10 07:54:01 k8s-msr-1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Jan 10 07:54:01 k8s-msr-1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 10 07:54:01 k8s-msr-1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 10 07:54:02 k8s-msr-1 kubelet[5151]: E0110 07:54:02.077945    5151 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"

Accessing Log Data - Control Plane​

  • Run as Pods
kubectl logs -n kube-system $POD_NAME
  • Run from systemd
systemctl status kubelet.service  # check service status
journalctl -u kubelet.service

Event​

全局 Event​

kubectl get events
kubectl get events
kubectl get events --field-selector type=Warning,reason=Failed
kubectl get events --watch &   ( run `fg` and ctrl +c to break it)

各資源 Event​

kubectl describe
kubectl describe pods nginx

JsonPath​

取得所有 Pod 的名稱
kubectl get pods -o jsonpath='{.items[].metadata.name}'
取得所有在使用的 Image 的名稱
kubectl get pods --all-namespaces -o jsonpath='{.items[].spec.containers[].image}'
取得所有在使用的 Image 的名稱，加入換行
kubectl get pods --all-namespaces -o jsonpath='{.items[].spec.containers[].image}{"\n"}'
建立 filter 在 log 之上
kubectl get nodes -o jsonpath="{.items[].status.addresses[?(@.type=='InternalIP')].address}"
排序 Log
kubectl get pods -A -o jsonpath='{.items[].metadata.name}{"\n"}' --sort-by=.metadata.name
取得 node 的 CPU 數量
kubectl get no -o jsonpath='{.items[].metadata.name} {"\n"} {.items[].status.capacity.cpu}'

k8s-msr-1 k8s-wrk-1 k8s-wrk-2
2 2 2
取得 node 的 CPU 與 Memory 數量
kubectl get no -o custom-columns=NODE:.metadata.name,CPU:status.capacity.cpu,MEMORY:.status.capacity.memory
NODE        CPU   MEMORY
k8s-msr-1   2     3998200Ki
k8s-wrk-1   2     3998200Ki
k8s-wrk-2   2     3998192Ki
kubectl get no -o custom-columns=<cloume-name>:<json path>

Centralized logging solutions​

ELK Stack